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ABSTRACT 

The National Advisory Cou ncil on Ed ucation 
Professions Development has been reviewing some of the major 
evaluations and evaluation research in the period 1966 to date, 
seeking to identify the salient issues concerning the evaluation of 
educational endeavors. This document is a' preliminary statement of 
the findings and recommendations. The interest in assessing 
effectiveness is not confined to education. The following 
developments show the important role assigned to evaluation of 
federal activities generally: a) increasingly, in new legislation 
enacted and in amendments to existing statutes, explicit provision 
for evaluation has been made; b) the executive branch's eppUasis on 
"cost-effectiveness" assumes the capacity on the part of the agencie 
to carry out appropriate and effective evaluation; c) thero has iaeen 
a very substantial increase in the funds allotted for evaluation in 
the executive branch. The importance that has been assigned to 
evaluation, and the influence that such studies have exerted to date 
prompts a question that must receive serious consideration: Is 
"evaluation," as an aspect of social science research, capaiile of 
responding adequately to the extraordinary demands placed upon it by 
the developments cited above? The council will continue to 
investigate this question, but it has concluded that there is a need 
for a full-r'cale examination of the evaluation procedures of federal 
programs in education. (JA) 
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PREFACE 



The National Advisory Council on Education Professions Development was 
established by the Congress in 1967 to review and evaluate the effective- 
ness of the operations of the Education Professions Developnient Act (Title 
V of the Higher Education Act), and of all other programs of the Federal 
government having to do with the training and development of educational 
personnel. These programs, which involve an annual outlay in exces.s of 
half a billion dollars, are administered by a variety of Federal agencies: 
the Department of Health, Education, and Welfare; the National Science 
Foundation; the Office of Economic Opportunity; National Institutes of 
Health; the Manpower Administration of the Department of Labor; the National 
Endowments for the Arts and for I he Humanities; and similar agencies. The 
members of this Council are appointed by the President. The Council submits 
its reports to the President and Congress. 

In January of 1972 this Council submitted a report to the President 
and to the Congress in which we examined the policies governing a number 
of Federal efforts designed to bring about improvements in the training 
and development of those educational personnel who staff the schools and 
colleges of the nation. 

In that report, entitled Windws to the Bureaucracy , we concluded 
that — with reference to policies needed to insure effective results 
from legislation ~ "the resources devoted to the formulation of poli- 
cies are inadequate; the processes employed are primitive; and the 
results are unacceptable." 

We outlined some specific steps that might be taken by both the Con- 
gress and the Executive Branch to bring about improvements in the formu- 
lation of "policies of effective means." 

Since that time we have been conducting an inquiry into another 
equally important subject: the way in which Federal programs in education -- 
and the state and local projects which are supported with these program 
funds— are being evaluated . The focus has been on the training of edu- 
cational personnel, though we have included in our concerns the evaluation 
of other aspects of education. 

The Council has addressed the question of evaluation on two earlier 
occasions. The first was in the Council's Second Annual Report, submitted 
in January 1969. This report ranged over a number of issues bearing on 
the improvement of the Federal effort in education. One chapter was 
devoted to Research and Evaluation. 



Evaluation was the subject of a special report devoted exclusively 
to thifi topic. This was submitted in May 1969. We concluded this 
special report by saying: 



Id its second annual report, submitted earlier this 
year to the President and Congress, this Council 
devoted a major section to research and evaluation. 
It has expanded on its statements in that report at 
this time because the need for sensible policies 
concerninR evaluation of educational programs; is 
rapidly emerging as one of the malor issues i n 
Aasrlcan education , 
(emphasis added) 

In its current inquiry, this Council has been reviewing some of the 
major evaluations "^nd evaluation research reported on during the period 
J966 to date. It has ul"^- been seeking to identify the salient issues 
concerning the evaluation of educational endeavors. 

This report is a preliminary statement of our findings and 
recommendat ionj , 
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Over the past ten years some $25 billion dollars has 
been invested in Federal efforts to improve education in 
the elementary and secondary schools, in the colleges and 
universities, and in other settings. 

Tl^e early part of this period saw the Federal 
government initiate a range of actions designed to bring 
about improvements in almost every aspect of the educational 
system: a number of pieces of landmark legislation were 
passed; substantial funds were appropriated to carry out 
this legislation; programs to implement this legislation 
were developed in the Office of Education, the National 
Science Foundation, the Office of Economic Opportunity, the 
National Institutes of Health, the Manpower Administration 
of the Department of Labor, the National Foundation on the 
Arts and Humanities, and other agencies. 

More recently, the emphasis has shifted from the 
initiation of efforts aimed at bringing about Improvements 
in American education to an interest in determining whether 
these efforts are achieving their goals . 



-1- 



This interest in evaluating effectiveness is not 
confined to education. The following developments give 
some indication of the important role that has now been 
assigned to evaluation of Federal activities generally: 

1. Increasingly in new legislation enacted and in 
amendments to existing statutes explicit provision 
for evaluation has been made. The General 
Accounting Office reports that in the areas of 
health, education, welfare, economic opportunity, 
and like Federal activities, there are twenty- 
three Acts in which formal objective evaluation is 
now prescribed. Some of these statutes have 
numerous provisions for evaluation. 

2. The Executive Branch has been placing increasing 
emphasis on "cost-effectiveness" as a means for 
determining whether the results from given Federal 
programj are in proportion to the money expended. 
Such a procedure assumes the capacity on the part 
of the agencies to carry out appropriate and 
effective evaluation* 
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3. There has been a very substantial increase in the 
funds alloted for evaluation in the Executive 
Branch. The General Accounting Office reports that 
expenditures for non-defense program evaluation 
have jumped from $19 million in 1969 to $110 million 
in 1972. 

4. It is clear that before the end of this year the 
Congress will have instituted some major reforms 
related to fundamental aspects of its own procedures. 
We speak of provisions for the establishment of 
budget ceilings ; for machinery which wil!" enable 

the Congress to set broad areas of priority concern; 
and for the creation of a Congressional Office of 
the Budget. The success of these reforms will depend, 
in no small way, on the capacity of the Congress to 
evaluate the effectiveness of the programs it has 
legislated — or at least to interpret evaluations 
conducted by the Executive Branch and to appraise 
the quality of such evaluations. 
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In the field of education, the amount of activity 
devoted to evaluation has been considerable. Numerous 
evaluation studies, as well as books, monographs and 
articles analyzing the findings of these studies, have 
been published all in an attempt to determine what 
works and what does not work and, indeed, whether 
anything works in education. 

The importance that has been assigned to evaluation, 
and the influence that evaluation studies have exerted to 
date, prompt a question that must receive serious 
consideration : 

Is "evaluation", as an aspect of social 
science research, capable of responding 
adequately to the extraordinary demands 
placed upon it by the developments cited 
above ? 

This Council has been exploring this question over 
the last few months. This exploration has consisted of 
a review of major evaluation studies and an attempt to 
identify those salient issues which the President and the 
Congress might take into account in making decisions 
concerning the evaluation of educational effectiveness, 
and in appraising existing evaluations of Federal efforts 



in education. We intend to continue this inquiry. 
However, we feel it appropriate to report two conclusions 
arrived at up to this point: 

1. There is a need for a full-scale examination 
of the concepts, methods and manner of 
conducting evaluations of Federal programs 
in education. 

2, Federal agencies responsible for evaluating 
educational programs should prepare policy 
statements which identify and record the 
positions of these agencies on the major 
issues concerning evaluation. Such positions 
would form a basis for guiding the decisions 
of these agencies with respect to the kinds 
of evaluation they intend to pursue. 
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